Supporting Priorities and Improving Utilization of the IBM SP2 Scheduler Using Slack Based Backfilling

Running jobs on the IBM SP2, as in most distributed memory parallel system in the market today, is done by giving each job a subset of the available processors for its exclusive use. Scheduling jobs in FCFS order suffers from severe fragmentation that leads to utilization loss. This led Argonne National Lab, where the first large SP1 was installed, to develop the EASY scheduler, which has since then been adopted by many other SP2 sites. This scheduler uses aggressive backfilling: Small jobs may be moved way ahead in the queue, while large jobs may suffer an unbounded delay. A more conservative backfilling strategy, which retains the predictability feature of FCFS, seems to equalize EASY’s performance on average workloads. None of the above schedulers support prioritization – allowing the administrators or the users themselves to schedule a job with a high or low priority, subject to accounting considerations such as paying for priority, preferred groups, quotas, emergencies and so on. This paper presents a scheduler that supports both user and administrative priorities. The scheduler gives each waiting job a slack, which determines how long it may have to wait before running: ‘Important’ and ‘heavy’ jobs will have little slack in comparison with others. When a new job is submitted, all possible schedules are priced according to utilization and priority considerations and as long as no job is delayed beyond its slack, the cheapest schedule is chosen. This gives the scheduler the flexibility to effectively backfill, and preserves the bounded delay advantage of FCFS and conservative backfilling over EASY. Experimental results show that the priority scheduler reduces the average wait time by about 15% relative to EASY in an equal priorities scenario, and is responsive to differential priorities as well. Supporting Priorities and Improving Utilization of the IBM SP2 Scheduler Using Slack Based Backfilling